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ABSTRACT 


Thanks to an increase in the amount of information on the Internet and the spread of ICT-supported educational 
environments, much attention has been paid to learning support based on “smart” recommendation technologies. In this 
study, we propose an education improvement model based on the recommender system using the human-in-the-loop 
design strategy. Our proposed model enhances not only learners via recommendation, but also teachers and the system 
itself through the interaction between teachers and the system. In this paper, we introduce the details of the proposed 
model and implementation strategy followed by a report of preliminary experimental results. 


KEYWORDS 


Education Improvement, Recommendation, Learning Analytics, Learning Log 


1. INTRODUCTION 


The increasing amount of information on the Internet, that is, the high number of open knowledge objects 
such as Wikipedia, iTuneU, and repositories managed by universities all over the world seems to be useful 
for supporting learning. However, learners are often discouraged by the time spent and difficulty in finding 
suitable and useful knowledge online (e.g., news articles, academic papers, and blogs) from a mass of search 
results (Chen et al., 2014). To use the information effectively and efficiently, it is necessary to build a system 
for the appropriate use of this vast knowledge. The recommender system for learning support is a common 
and useful system in this era of vast information (e.g., Erdt et al., 2015). Thus, a recommender system 
provides an appropriate learning environment that helps learners find appropriate knowledge targets within 
the vast amount of information on the Internet. 

Researchers have been applying these resources to learning systems to clarify the effects of recommender 
systems for learning (e.g., Liang et al., 2006; Chen et al., 2007; Yamada et al., 2014). Many techniques for a 
recommender system have also been proposed. In particular, there are a large number of studies aimed at 
supporting learners based on student attributes such as learning situation (e.g., Tarus et al., 2018), preference 
based on learning logs (e.g., Salehi et al., 2014), search queries (e.g., Mbipom et al., 2018), and so on. On the 
other hand, there are a few studies that try, not only to support learners, but also to improve teaching 
materials and lectures. In addition, many existing studies unilaterally recommend online resources 
determined by the system, and do not design a feedback loop to improve the system itself. In order for the 
system to recommend appropriate resources from an educational viewpoint and for individual learners, it is 
desirable to involve teachers in the loop of recommendation and improvement of education and learning. In 
this paper, we propose an education improvement model, which collaborates with the page-wise 
recommendation method (Nakayama et al., 2019). The page-wise recommendation system provides 
recommended information for each page as shown in Figure 1. The original recommender system has some 
drawbacks (see the next section for the details), so our proposed model involves teachers to complement the 
drawbacks. In addition, the teachers can also get feedback from the system to improve their teaching and 
teaching materials. Therefore, our proposed model enhances not only learners via recommendation, but also 
the teachers and the system based on the human-in-the-loop design strategy. In the following section of this 
paper, we introduce the details of the proposed model and implementation strategy followed by a report of 
preliminary experimental results. 
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Figure 1. Page-wise recommendation system 


2. EDUCATION IMPROVEMENT MODEL BASED ON PAGE-WISE 
RECOMMENDATION 


In this study, we propose an education improvement model based on a system that recommends 
supplementary materials related to teaching materials used in lectures. The overview of the system 
configuration is shown in Figure 2. The proposed model improves education and learning by cycling through 
the following process: 

(1) A teacher prepares a lecture plan and teaching material in preparation for a lecture and registers the 
teaching material in an e-Book system. 

(2) The proposed system selects recommended information related to the teaching material from the 
Internet. 

(3) The e-Book system provides the recommendation to the students during the lecture or self-study. 

(4) The system analyzes the learning activity logs and gives feedback to the teacher. 
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Figure 2. Overview of education improvement model 
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A method to calculate recommended pieces of information in (2) is inspired by an existing study 
proposed by Nakayama et al (2019). According to the survey in the paper, the recommended items were 
mostly suitable for the content in the lecture materials. However, it sometimes contained inappropriate 
information for learning because the recommended items were automatically retrieved from the Internet and 
unilaterally provided to students. The problem is that it is not possible to know whether the recommended 
information is appropriate from an educational viewpoint or for individual learners. Therefore, it is necessary 
to have a mechanism for acquiring feedback from each user including the teachers and students. In order to 
solve this problem, in this study, a teacher intervenes in the selection of recommended information in (2). 
The system provides a list of candidates for recommended items on a web page, and the teacher investigates 
and decides whether each one is suitable for recommendation or not. We believe that teachers! intervention in 
selecting recommended information has several advantages for students and teachers, as listed below: 

@ We can prevent undesirable situations in which inappropriate or unnecessary information is 
recommended to students. 

@ The selection results by the teacher are helpful for the system to improve the automatic suggestion of 
recommended information. 

The recommendation results for students are analyzed in (4) based on the student learning activity log 
(click on the link to the recommended information) in the e-Book system. The analysis results are feedback 
to teachers, students, and the system itself. Recommending information corresponding to each page of 
teaching material not only facilitates students’ learning, but also allows teachers to understand the students’ 
reaction to teaching materials used in lectures and proceed with a lecture. Therefore, teachers can use the 
analysis results as feedback from students to improve teaching materials and proceed with a lecture. The 
responses from students are also meaningful data that can be used to investigate whether recommended 
information was useful or not. The operation logs of clicking the recommended information indicates 
whether the student has interest in the recommended information. By collecting the access information of 
recommended information from each student, it will become possible to provide personalized recommended 
information according to the individual's learning level in the future. In this way, by examining and 
evaluating the recommended information from both teachers and students, it is possible to make high-quality 
recommendations, and further education improvement can be achieved. We will describe the system to 
realize this education improvement model in the following section. 


3. IMPLEMENTATION 


3.1 System Configuration 


We use two systems to recommend information related to the teaching materials treated in lectures. One is an 
e-Book system called BookRoll (Ogata, 2015, 2017), the other is Knowledge extension based on Tailor-made 
Information Provision System (K-TIPS). 

BookRoll is an e-Book viewer which has three databases. In each database, teaching materials that are 
e-textbook used in lectures, recommended information for each piece of teaching materials, and student’s 
activity logs to e-Book system are stored. A teacher conducts a lecture using an e-textbook stored in the 
e-Book system and students learn by using it during and outside a lecture. Various functions are implemented 
in the e-Book system, and a recommendation function is one of them. Figure 3 shows the user interface of the 
e-Book viewer. Students can access the recommended items by clicking the upper right button of the viewer 
as necessary. In addition, each student can submit a response about whether he/she could understand the 
contents on the page or not, by clicking the green (understood) or red (not understood) button. When a 
student uses any of the functions of the e-Book system (such as open an e-textbook, go to the next page, 
highlight, click a link to a recommended article, etc.), learning activity logs are automatically recorded in the 
database. 
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(A) Material viewer (B) Recommendation viewer 
Figure 3. Viewer of e-Book system 


The K-TIPS is a web application that displays candidates for recommended information, and allows 
teachers to select recommended information related to each page of teaching material. In this system, the 
teaching material designated by the teacher is analyzed, and the recommended information is automatically 
selected from the Internet and presented to the teacher. Then, the teacher investigates the candidates of 
recommended information and decides the information to be recommended to the student. Finally, it is 
registered in the e-Book system by the K-TIPS. The details of the web application are explained in the 
following sections. 


3.2 K-TIPS 


3.2.1 Overview of K-TIPS 


In this research, as mentioned in section 2, we believe that there are various advantages regarding the 
teacher's intervention in selecting recommended information. Therefore, we developed a web application for 
teachers to investigate the automatically selected recommended information from the Internet. The interface 
of the K-TIPS is shown in Figure 4. 


(A) Selection screen (B) Investigation screen (C) Confirmation screen 
Figure 4. Interface of K-TIPS 


Figure 4-(A) is a web page for selecting lecture materials to be analyzed for calculating recommended 
information. In this page, the teacher specifies the maximum number of recommended items per page. When 
the specification is complete, they click a green button. Then, the recommended information for the teaching 
material is automatically selected from the Internet and displayed on the screen. The automatic selection 
method of recommended information is described in the next section. 

Figure 4-(B) is a web page for investigation. On this page, two details about each recommended item are 
displayed in a table format: the page number recommended in the e-Book system and the title of the 
recommended item. In addition, a check box is displayed at the left end of the table. A link to the 
recommended information is attached to the displayed title, and the teacher can confirm the content of the 
recommended item by clicking the title. If the teacher judges that the content of the recommended 
information is not appropriate, they can uncheck the box. After completing the investigation of all candidates, 
they click the green registration confirmation button to proceed to the confirmation page. 
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Figure 4-(C) is a web page to confirm the recommended information registered in the e-Book system. In 
this page, recommended items with boxes that are still checked after investigation are displayed, and 
recommended items with boxes that are unchecked are displayed as "deleted." The teacher verifies whether 
the recommended information to be registered is correct, and if there is some mistake, they can return to the 
investigation page again by clicking the return button. If there are no mistakes, the recommended information 
is registered in the e-Book system by clicking the registration button, and it is then recommended to the 
student. 

This system stores a list of recommended items automatically selected from the Internet before 
investigation. Therefore, it is possible to analyze the recommended items deleted by teachers by comparing 
them with a list of recommended items registered in the e-Book system after investigation. The analysis 
results can be used to improve the accuracy of the automatic selection method of recommended information, 
and it is possible to support education and learning by providing recommendations that are more suitable for 
learning. 


3.2.2 Automatic Selection Method of Recommended Information 


In this section, we give an overview of the automatic selection method of recommended information for each 
page of a teaching material (Nakayama et al., 2019). In this method, to select recommended information, the 
method takes three analytical steps, as follows: 

1. Word extraction from teaching material; 

2. Calculation of the importance of extracted words; 

3. Retrieval of recommended information from the Internet based on important words. 

First, nouns in the teaching material are extracted. In the e-Book system, teaching materials are stored as 
PDFs, and text information about sentences in registered teaching materials is extracted from the PDFs 
automatically. Therefore, the text information can be utilized to extract sentences and words from the 
teaching material. In this research, we focus on Japanese teaching materials. In contrast to languages 
segmented by spaces such as English, Japanese is a non-segmented language. Therefore, it is necessary to 
divide the sentences to extract words. In this study, we apply MeCab morphological analysis (Kudo et al., 
2004). Morphological analysis is the technique of dividing natural language into morphemes, the smallest 
units of words. In this method, proper nouns are extracted from a sentence through morphological analysis. 
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Figure 5. Overview of calculating importance of extracted words 


Second, the importance of the extracted words on each page are estimated to select recommended 
information for each page of the teaching material. The estimation procedure involves two steps: 1) dividing 
each page of a teaching material into a subset of pages (refer to them as “segments”) in terms of similar 
topics, and 2) calculating the importance of the words on each page within each segment. Figure 5 shows an 
overview of the process flow to estimate the importance of extracted words. In this method, whole pages of 
teaching material are divided into segments according to the topic of the contents because it may be 
considered that there is a difference in importance depending on the topic, even for the same word. After 
dividing into subsets of pages, the importance of each word is calculated for each segment. The importance 
of words in each segment is estimated using the TF-IDF method (Salton & Buckley., 1988). The TF-IDF 
method is a method for evaluating the importance of words in a document using two indicators: term 
frequency (TF) and inverse document frequency (IDF). Teaching materials are divided into the title area and 
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the body area. The title represents the contents of each page, and words in the title area seem to be important. 
Therefore, the importance of each word is calculated by weighing words in the title area to the importance of 
the TF-IDF method. 

Third, recommended information is retrieved from the Internet based on the importance of the word. First, 
let the top n words of importance in each page be the important words representing the content of the page. 
Next, the top n words are used as the search query to retrieve related websites. Then, the top m of the search 
results are selected to be the recommended items of information. 


3.3 Student Activity Log 


An analysis of recommendation results in (4) in section 2 is performed by analyzing student activity logs 
collected by the e-Book system. As mentioned above, various functions are implemented in the e-Book 
system. For example, adding a memo or a marker to a current page, evaluating whether a student understands 
the current page, accessing the current page's recommended information, etc. As shown in Figure 6, various 
activity logs of students using such functions can be collected. 

An analysis of recommendation results uses logs for recommended information (a log with the operation 
name “CLICK RECOMMENDATION” in Figure 6). From the log data of recommendations, it is possible to 
identify the user who accessed the recommended information, the page number of a teaching material, and 
which recommended information is accessed. Utilization of these learning activity logs has the possibility to 
improve education and learning in terms of following aspects: 

@ Update the teaching material by analyzing the number of clicks in each page of the teaching material. 

@ Determine the demand of each student for recommended information. 

@ = Realize personalized recommendation. 

Moreover, we believe that it is possible to analyze from other various viewpoints by utilizing the log data 
other than the recommendation. For example, it is possible to analyze the relationship between the reaction to 
the recommendation and the student's level of understanding by using the log indicating whether the student 
could understand the content of the current page (a log with the operation name “GETIT” or “NOTGETIT” 
in Figure 6). As a result, we believe that the recommendation results can be used to further improve 
education. 


User ID e-book ID e-book title Page Operation Description .. Time 

XXXXXXXXXKX COCCCCCCG D-D6-iII{® 14 CLICK_RECOMMENDATION = https://*******"*** | 2019-07-01 11:26:15 
woowea00ax ococcccc0e0 D-06-((R 13° NEXT «» 2019-07-01 11:26:17 
XXXXXIXXXKXX OOCOCd000 D-06-iEi{ 15 NOTGETIT .. 2019-07-01 11:26:20 
XXXXXXXXXXXX COC000000 D-06-iI5){® 16 CLICK_RECOMMENDATION i https://*********** |. 2019-07-01 11:26:25 
KOOUOKXKKAX OCoCdd000 D-06-/5{R 17 GETIT «. 2019-07-01 11:26:31 


Figure 6. Samples of student activity logs stored in the e-Book system 


4. EXPERIMENTS 


We have been conducting experiments using the proposed system in multiple lectures for first- and 
second-year college students using the proposed system since April 2019. A total of 98 teaching materials 
used in 16 courses in the first semester of the 2019 school year at Kyushu University were used for the 
experiments and recommended information was provided to students by the end of July 2019. As a result of 
the recommendations, 1061 activity log data for recommendations were collected from 247 students. In this 
study, we have developed a web application for teachers to confirm the usage status of recommended 
information. Figure 7 is the display screen of the recommendation result. The teacher can confirm how many 
times the recommended items were clicked by students in each page. In the graph displayed, the x-axis is the 
page number of the teaching material, and the y-axis is the number of clicks for the recommended items of 
information. In addition, if the teacher moves the mouse cursor to the bar chart, its corresponding page is 
displayed below the graph. If the number of clicks for the recommended items is extremely high compared to 
other pages, it is taken into consideration that there may be a problem with the contents of the teaching 
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materials or lectures. It has shown that the number of clicks on the relevant page may increase if the content 
of the teaching material is difficult or if there is a problem with the style of the lecture (Nakayama et al., 
2019). In this way, utilizing the recommendation results can be used to improve education. 
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Figure 7. The display screen of the recommendation results 


Also, investigation by teachers has been implemented since July 2019. By the end of July 2019, 
investigation was conducted by a total of four teachers for the recommended information on 42 teaching 
materials. As a result of the investigation, 1.4% of the recommended information selected automatically was 
judged as inappropriate or unnecessary for learning and was deleted by the teacher. We analyzed the titles 
and URLs of recommended information deleted by investigation. As a result, we saw that there were many 
words related to games (such as “games,” “gamer,” “game design,” etc.) and words related to products (such 
as “Amazon,” “product,” and “shohin,” which in Japanese means product, etc.) in the recommended 
information. The reason why the recommended information about the game was presented to the students is 
that an educational information engineering course included a class about game-based learning. In addition to 
this, many keywords such as “syllabus” and “yahoo” that were considered unnecessary for learning were 
confirmed. Based on the results of these investigations, we can implement high-quality recommendations for 
learning by designing a mechanism for automatically excluding information inappropriate for learning. In 
addition, we confirmed cases where all recommended items on some pages were deleted by investigation. It 
can be inferred that these were pages that did not require a recommendation (such as a page with notes about 
the lecture, etc.). It may be possible to identify pages that do not require a recommendation by using the 
investigation result. 


5. CONCLUSION 


In this paper, we proposed an education improvement model based on a system that recommends 
supplementary teaching materials related to teaching materials used in lectures. Then, we described the 
system for realizing the proposed education improvement and reported on the current status of 
recommendation experiments conducted in the university environment. We believe that the proposed system 
enables both teachers and students to improve their teaching and learning experiences through interaction 
with recommended information. A page in which many students used the recommended information suggests 
the possibility that the contents of the material are difficult to understand, so that a teacher will consider an 
update of the contents. 

In future work, as mentioned above, we will work on the development of the method to calculate 
recommended information using the investigation result by the teachers and the learning support by the 
individual recommendation from the recommendation result to the student. In addition to these, we will also 
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examine information recommendation for summarized teaching materials. The summarized teaching material 
is a collection of pages containing important content extracted from the teaching material (Shimada et al., 
2017). The contents of the teaching material are automatically summarized to a small number of pages, so 
that only the essential points are given to learners. The combination of summarization technique and our 
recommender system will help learners understand the summary of lecture contents and present an 
opportunity for knowledge extension by using recommended information. We are sure that the collaboration 
with other digital technologies will support learning and teaching in a new digital educational era. 
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